What to Fix? Distinguishing between design and non-design rules in automated tools

Technical debt---design shortcuts taken to optimize for delivery speed---is a critical part of long-term software costs. Consequently, automatically detecting technical debt is a high priority for software practitioners. Software quality tool vendors have responded to this need by positioning their tools to detect and manage technical debt. While these tools bundle a number of rules, it is hard for users to understand which rules identify design issues, as opposed to syntactic quality. This is important, since previous studies have revealed the most significant technical debt is related to design issues. Other research has focused on comparing these tools on open source projects, but these comparisons have not looked at whether the rules were relevant to design. We conducted an empirical study using a structured categorization approach, and manually classify 466 software quality rules from three industry tools---CAST, SonarQube, and NDepend. We found that most of these rules were easily labeled as either not design (55%) or design (19%). The remainder (26%) resulted in disagreements among the labelers. Our results are a first step in formalizing a definition of a design rule, in order to support automatic detection.Comment: Long version of accepted short paper at International Conference on Software Architecture 2017 (Gothenburg, SE

A Study of Documentation for Software Architecture

Documentation is an important mechanism for disseminating software architecture knowledge. Software project teams can employ vastly different formats for documenting software architecture, from unstructured narratives to standardized documents. We explored to what extent this documentation format may matter to newcomers joining a software project and attempting to understand its architecture. We conducted a controlled questionnaire-based study wherein we asked 65 participants to answer software architecture understanding questions using one of two randomly-assigned documentation formats: narrative essays, and structured documents. We analyzed the factors associated with answer quality using a Bayesian ordered categorical regression and observed no significant association between the format of architecture documentation and performance on architecture understanding tasks. Instead, prior exposure to the source code of the system was the dominant factor associated with answer quality. We also observed that answers to questions that require applying and creating activities were statistically significantly associated with the use of the system's source code to answer the question, whereas the document format or level of familiarity with the system were not. Subjective sentiment about the documentation format was comparable: Although more participants agreed that the structured document was easier to navigate and use for writing code, this relation was not statistically significant. We conclude that, in the limited experimental context studied, our results contradict the hypothesis that the format of architectural documentation matters. We surface two more important factors related to effective use of software architecture documentation: prior familiarity with the source code, and the type of architectural information sought.Comment: accepted to EMSE

Registered Reports in Software Engineering

Registered reports are scientific publications which begin the publication process by first having the detailed research protocol, including key research questions, reviewed and approved by peers. Subsequent analysis and results are published with minimal additional review, even if there was no clear support for the underlying hypothesis, as long as the approved protocol is followed. Registered reports can prevent several questionable research practices and give early feedback on research designs. In software engineering research, registered reports were first introduced in the International Conference on Mining Software Repositories (MSR) in 2020. They are now established in three conferences and two pre-eminent journals, including Empirical Software Engineering. We explain the motivation for registered reports, outline the way they have been implemented in software engineering, and outline some ongoing challenges for addressing high quality software engineering research.Comment: in press as EMSE J. commen

Cross-Dataset Design Discussion Mining

Being able to identify software discussions that are primarily about design, which we call design mining, can improve documentation and maintenance of software systems. Existing design mining approaches have good classification performance using natural language processing (NLP) techniques, but the conclusion stability of these approaches is generally poor. A classifier trained on a given dataset of software projects has so far not worked well on different artifacts or different datasets. In this study, we replicate and synthesize these earlier results in a meta-analysis. We then apply recent work in transfer learning for NLP to the problem of design mining. However, for our datasets, these deep transfer learning classifiers perform no better than less complex classifiers. We conclude by discussing some reasons behind the transfer learning approach to design mining.Comment: accepted for SANER 2020, Feb, London, ON. 12 pages. Replication package: https://doi.org/10.5281/zenodo.359012

Supporting requirements model evolution throughout the system life-cycle

Requirements models are essential not just during system implementation, but also to manage system changes post-implementation. Such models should be supported by a requirements model management framework that allows users to create, manage and evolve models of domains, requirements, code and other design-time artifacts along with traceability links between their elements. We propose a comprehensive framework which delineates the operations and elements necessary, and then describe a tool implementation which supports versioning goal models